System for online automated exam proctoring

ABSTRACT

Certain embodiments may be directed to a system and method for proctoring an exam, and more particularly, for online automated exam proctoring. A method may include checking a test-taker&#39;s computing device for compatibility, and for content that provide unauthorized aid to the test-taker during a testing session. The method may also include taking a photo of the test-taker, recording the test-taker by performing a room pan while analyzing the surrounds of the test-taker for unauthorized objects, and validating the identity of the test-taker by way of at least one of presenting challenge questions, obtaining voice biometrics, or obtaining keystroke biometrics. The method may further include recording the test-taker&#39;s audio or video and desktop feed, determining if the test-taker is exhibiting questionable behavior, and suspending the testing session if it is determined that the test-taker is exhibiting questionable behavior.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation-in-Part of U.S. Non-Provisionalpatent application Ser. No. 15/462,676, filed on Mar. 17, 2017, which isa continuation of U.S. Non-Provisional patent application Ser. No.14/067,796, filed on Oct. 30, 2013, now U.S. Pat. No. 9,601,024, whichis a continuation of U.S. Non-Provisional application Ser. No.13/007,341, filed on Jan. 14, 2011, now abandoned, which claims thebenefit of U.S. Provisional Patent Application No. 61/295,508 filed onJan. 15, 2010. The entire contents of all of these earlier filedapplications are hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

Certain embodiments of the invention is directed to a system and methodfor proctoring an exam, and more particularly, for online automated examproctoring.

BACKGROUND OF THE INVENTION

Since the invention of the classroom, their have been exams to determinehow well a student has learned the subject matter. With the advent oflarge classes, and written exams, the monitoring (proctoring) of examshas been a necessity. As well known in the art, teachers, and then laterprofessional proctors, remain in the classroom to observe the examtaking procedure to prevent false results through cheating. This inperson proctoring of exams has been satisfactory for use in conventionaleducation environments. However, the advent of distance learning byeducational institutions such as Andrew Jackson University, theUniversity of Phoenix and others have made it impracticable topersonally monitor each student taking an exam because without thecommonality of the physical classroom, the students are too widelydisbursed.

Many institutions and companies faced with this problem have attemptedto circumvent it by diluting the course requirements and eliminating therequirement of a proctored exam in lieu of written papers or projects.Those institutions who had tried to solve the problem approached thesituation with a high level of automation and the lowest level of humaninteraction possible making the system rife for fraud.

Further, since the invention of remote online proctoring, there havebeen inherent challenges that needed to be overcome in order to achievesuccess in replicating and replacing the traditional face-to-faceproctoring method. One such challenge found in both traditionalface-to-face proctoring and online proctoring is human error. Regardlessof the selection process, training, auditing, and oversight that mightbe applied to a proctor, humans will inevitably continue to error.

Another challenge has been inefficiency. For instance, a highly trainedproctor is only able to accurately watch a maxim of six to eightsimultaneous online exam sessions using current industry accepted onlineproctoring methods. A further challenge is scalability. For example, ifan organization were to proctor only two of the major college boundentrance exams, approximately 4 million exams in a year (uniformlydistributed across 365 days and the exams were 2 hours long), thatorganization would need to employ approximately 642 proctors to overcomethat volume. That requirement is for two exams with perfect examdistribution across a year. In reality, there are hundreds of thousandsof different exams with hundreds of million exams proctored annually.

Customer service has also been a challenge. Since people tend to beaverse to waiting, one of the major contributors to customersatisfaction in online proctoring the wait time. Wait time includes thetime it takes to connect to a remote proctor, authenticate thetest-taker, secure their computing environment, and get them into theirexam. Further, bandwidth availability and stability have become majorissues in online proctoring.

Accordingly, a system and method for overcoming the shortcomings of theprior art by enabling a live proctor to monitor a student's examprocess, including student identification, as well as the immediate examenvironment is desired. It is also desired to consider user experience,cheater adaptability, and hybrid scalability.

SUMMARY OF THE INVENTION

According to certain embodiments, an apparatus may include at least oneprocessor, and at least one memory including computer program code. Theat least one memory and the computer program code may be configured to,with the at least one processor, cause the apparatus at least to check atest-taker's computing device for compatibility, and for content thatprovide unauthorized aid to the test-taker during a testing session. Theat least one memory and the computer program code may also be configuredto, with the at least one processor, cause the apparatus at least totake a photo of the test-taker. In addition, the at least one memory andthe computer program code may be configured to, with the at least oneprocessor, cause the apparatus at least to record the test-taker byperforming a room pan while analyzing the surrounding of the test-takerfor unauthorized objects. The at least one memory and the computerprogram code may further be configured to, with the at least oneprocessor, cause the apparatus at least to validate the identity of thetest-taker by way of at least one of presenting challenge questions,obtaining voice biometrics, or obtaining keystroke biometrics. The atleast one memory and the computer program code may also be configuredto, with the at least one processor, cause the apparatus at least torecord the test-taker's audio or video and desktop feed. Further, the atleast one memory and the computer program code may be configured to,with the at least one processor, cause the apparatus at least todetermine if the test-taker is exhibiting questionable behavior based onan analysis of the compatibility of the test-taker's computing device,the test-taker's photo, the room pan, validation of the test-taker'sidentity, and recordation of the test-taker's audio or video and desktopfeed. In addition, the at least one memory and the computer program codemay be configured to, with the at least one processor, cause theapparatus at least to suspend the testing session if it is determinedthat the test-taker is exhibiting questionable behavior.

The at least one memory and the computer program code may also beconfigured to, with the at least one processor, cause the apparatus atleast to provide the test-taker with test rules defined by aninstitution of the test-taker, and check the test-taker's computingdevice for applications, code, or materials used to provide unauthorizedaid during the testing session. In addition, the at least one memory andthe computer program code may be configured to, with the at least oneprocessor, cause the apparatus at least to send the test-taker a warningsignal when it is determined that the test-taker is exhibitingquestionable behavior, and communicate with a live proctor when it isdetermined that the test-taker is exhibiting questionable behavior.Further, the at least one memory and the computer program code may beconfigured to, with the at least one processor, cause the apparatus atleast to adjust system settings for proctored test sessions, reviewcompleted proctoring test sessions, and view sessions flagged forquestionable behavior of the test-taker.

According to certain other embodiments, a method may include checking atest-taker's computing device for compatibility, and for content thatprovide unauthorized aid to the test-taker during a testing session. Themethod may also include taking a photo of the test-taker, recording thetest-taker by performing a room pan while analyzing the surrounding ofthe test-taker for unauthorized objects, and validating the identity ofthe test-taker by way of at least one of presenting challenge questions,obtaining voice biometrics, or obtaining keystroke biometrics. Themethod may further include recording the test-taker's audio or video anddesktop feed, and determining if the test-taker is exhibitingquestionable behavior based on an analysis of the compatibility of thetest-taker's computing device, the test-taker's photo, the room pan,validation of the test-taker's identity, and recordation of thetest-taker's audio or video and desktop feed. Further, the method mayinclude suspending the testing session if it is determined that thetest-taker is exhibiting questionable behavior.

The method may further include providing the test-taker with test rulesdefined by an institution of the test-taker, checking the test-taker'scomputing device for applications, code, or materials used to provideunauthorized aid during the testing session, sending the test-taker awarning signal when it is determined that the test-taker is exhibitingquestionable behavior, and communicating with a live proctor when it isdetermined that the test-taker is exhibiting questionable behavior. Inaddition, the method may include adjusting system settings for proctoredtest sessions, reviewing completed proctoring test sessions, and viewingsessions flagged for questionable behavior of the test-taker.

According to certain other embodiments, a computer program may beembodied on a non-transitory computer readable medium. The computerprogram, when executed by a processor, may cause the processor to checka test-taker's computing device for compatibility, and for content thatprovide unauthorized aid to the test-taker during a testing session. Thecomputer program, when executed by the processor, may also cause theprocessor to take a photo of the test-taker, record the test-taker byperforming a room pan while analyzing the surrounding of the test-takerfor unauthorized objects, and validate the identity of the test-taker byway of at least one of presenting challenge questions, obtaining voicebiometrics, or obtaining keystroke biometrics. The computer program,when executed by the processor, may further cause the processor torecord the test-taker's audio or video and desktop feed, and determineif the test-taker is exhibiting questionable behavior based on ananalysis of the compatibility of the test-taker's computing device, thetest-taker's photo, the room pan, validation of the test-taker'sidentity, and recordation of the test-taker's audio or video and desktopfeed. In addition, the computer program, when executed by the processor,may cause the processor to suspend the testing session if it isdetermined that the test-taker is exhibiting questionable behavior.

Further, the computer program, when executed by the processor, may causethe processor to provide the test-taker with test rules defined by aninstitution of the test-taker, to check the test-taker's computingdevice for applications, code, or materials used to provide unauthorizedaid during the testing session, send the test-taker a warning signalwhen it is determined that the test-taker is exhibiting questionablebehavior, and communicate with a live proctor when it is determined thatthe test-taker is exhibiting questionable behavior. In addition, thecomputer program, when executed by the processor, may cause theprocessor to adjust system settings for proctored test sessions, reviewcompleted proctoring test sessions, and view sessions flagged forquestionable behavior of the test-taker.

Additional features, advantages, and embodiments of the invention areset forth or apparent from consideration of the following detaileddescription, drawings and claims. Moreover, it is to be understood thatboth the foregoing summary of the invention and the following detaileddescription are exemplary and intended to provide further explanationwithout limiting the scope of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate preferred embodiments of theinvention and together with the detailed description serve to explainthe principles of the invention. In the drawings:

FIG. 1 is a schematic view of a system for providing remote proctoringin accordance with the invention.

FIG. 2 is an example of a log in portal for use by the exam takingstudent in accordance with the invention.

FIG. 3 is an example of a scheduling screen presented in accordance withthe invention.

FIG. 4 is an exemplary screen shot of a reservation interface inaccordance with the invention.

FIG. 5 is a scheduling portal with countdown clock in accordance withthe invention.

FIG. 6 is an exemplary screen shot showing the beginning of an exam.

FIG. 7 is a flow diagram of the operation of the system in accordancewith the invention.

FIG. 8 is an exemplary screen shot of the interaction between a proctorand a student in accordance with the invention.

FIG. 9 is a screen shot of a collection of proctor reports in accordancewith the invention.

FIG. 10 is a screen shot for setting the parameters of the exam throttlein accordance with the invention.

FIG. 11 is an exemplary screen shot of the administrator screen showingthe administration schedule in accordance with the invention.

FIG. 12 is an exemplary screen shot enabling student to schedule an examin accordance with the invention.

FIG. 13 is a screen shot of an interactive page for a school report inaccordance with the invention.

FIG. 14 is a screen shot showing the manner in which a proctor accessesan appointment in accordance with the invention.

FIG. 15 is an exemplary screen shot of an edit menu to be utilized by aproctor in accordance with the invention.

FIG. 16 is an exemplary review of a contact window for the proctor toaccess contact info for a particular student in accordance with theinvention.

FIG. 17 is an exemplary screen shot of a proctor's administrativeconsole in accordance with the invention.

FIG. 18 is an exemplary screen shot enabling an administrator or proctorto edit or add users to the system in accordance with the invention.

FIG. 19 is an exemplary screen shot of an administration console to beutilized by the institution providing the exam to be proctored inaccordance with the invention.

FIG. 20 is an exemplary screen shot enabling the setting by the schoolof the exam parameters in accordance with the invention.

FIG. 21 is an exemplary screen shot of a cancellation report inaccordance with the invention.

FIG. 22 illustrates a system according to certain embodiments.

FIG. 23 illustrates a user flow and interaction model according tocertain embodiments.

FIG. 24 illustrates a partial test-taker user flow according to certainembodiments.

FIG. 25 illustrates a proctor user flow according to certainembodiments.

FIG. 26 illustrates a test-taker user flow according to certainembodiments.

FIG. 27 illustrates a continuation of the test-taker user flow of FIG.26 according to certain embodiments.

FIG. 28 illustrates an instructor or exam administrator user workflowaccording to certain embodiments.

FIG. 29 illustrates an architecture overview according to certainembodiments.

FIG. 30 illustrates a first server infrastructure architecture accordingto certain embodiments.

FIG. 31 illustrates a second server infrastructure architectureaccording to certain embodiments.

FIG. 32 illustrates a third server infrastructure architecture accordingto certain embodiments.

FIG. 33 illustrates a supervised machine learning flow according tocertain embodiments.

FIG. 34 illustrates an exemplary system according to certainembodiments.

FIG. 35 illustrates a flow diagram according to certain embodiments.

DETAILED DESCRIPTION OF THE INVENTION

The features, structures, or characteristics of the invention describedthroughout this specification may be combined in any suitable manner inone or more embodiments. For example, the usage of the phrases “certainembodiments,” “some embodiments,” or other similar language, throughoutthis specification refers to the fact that a particular feature,structure, or characteristic described in connection with the embodimentmay be included in at least one embodiment of the present invention.

Thus, appearances of the phrases “in certain embodiments,” “in someembodiments,” “in other embodiments,” or other similar language,throughout this specification do not necessarily all refer to the samegroup of embodiments, and the described features, structures, orcharacteristics may be combined in any suitable manner in one or moreembodiments. Additionally, if desired, the different functions discussedbelow may be performed in a different order and/or concurrently witheach other. Furthermore, if desired, one or more of the describedfunctions may be optional or may be combined. As such, the followingdescription should be considered as merely illustrative of theprinciples, teachings and embodiments of this invention, and not inlimitation thereof.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized in, with, or in conjunction with componentsand/or certain embodiments of the invention described herein. Thecomputer-usable or computer-readable medium may be, for example but notlimited to, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, device, or propagation medium. Morespecific examples (a non-exhaustive list) of the computer-readablemedium may include the following: an electrical connection having one ormore wires, a portable computer diskette, a hard disk drive (HDD), arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CDROM), an optical storagedevice, a transmission media such as those supporting the Internet or anintranet, or a magnetic storage device. A computer-usable orcomputer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer-usable medium may include a propagated data signal with thecomputer-usable program code embodied therewith, either in baseband oras part of a carrier wave. The computer usable program code may betransmitted using any appropriate medium, including but not limited towireless, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out methods of certain embodiments ofthe present invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

These computer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart(s) and/orblock diagram block or blocks. These computer program instructions mayalso be stored in a computer-readable medium that can direct a computeror other programmable data processing apparatus to function in aparticular manner, such that the instructions stored in thecomputer-readable medium produce an article of manufacture includinginstruction means which implement the function/act specified in theflowchart(s) and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart(s) and/or block diagram block or blocks.

Certain embodiments described herein may work with a large majority ofusers without the need for different hardware, complex softwareinstallations, and drastic modifications of testing behavior orenvironment. Further, accuracy of certain embodiments described hereinmay be improved by implementing machine learning to account for cheateradaptability. Other embodiments may be designed to operate alone, orwork with human counterparts to augment their ability to perform variousproctoring functions.

Certain embodiments may include a system that integrates with onlinetest delivery or online learning management systems to automaticallyproctor test-takers during their exams. The test-taker interface of thesystem may rely on the use of the test-taker's camera, microphone,keyboard, and mouse to collect behavioral input during the exam session.Based on the test-taker's collected behavioral input, the system mayflag or end the exam session and subsequently notify the exam providerin the event of aberrant behavior. According to other embodiments, thesystem may perform this analysis and determination of aberrant behaviorthrough the use of supervised machine learning software implemented on acomputing device.

According to certain embodiments, the test-taker's behavioral input maybe collected automatically by the test-taker's computer and sent back tothe media server for analysis. In certain embodiments, the collectedinput may be controlled by a programmed server, such as, for example amedia server described herein, which may have the ability to processbehavioral input. The media server may receive and evaluate this inputmany times a second. In addition, according to other embodiments, aprocessing server may also receive and evaluate this input many times asecond as with the media server.

In certain embodiments, the behavioral input may include, but notlimited to, facial characteristics, movement of the test-taker, soundsin the test-taker's environment, lighting of the test-taker'senvironment, and the test-taker's computer input from a keyboard and/ormouse clicks. According to certain embodiments, the inputs may beevaluated separately and in any combination. For example, in certainembodiments, it is possible to detect if someone, such as thetest-taker, is using external resources by looking for a loss of face ora profile view of the face of the test-taker for approximately 10 ormore seconds, and then followed by mouse and/or keyboard input(s) by thetest-taker, and then a subsequent answer submission by the test-taker.In other embodiments, the evaluation time may be varied, and may be, forexample, less than 10 seconds.

According to certain embodiments, biometrics of the test-taker may beestablished prior to the test, and then continuously monitored duringthe exam. The biometrics obtained from the test-taker may include, butnot limited to, voice, keystroke, and identity information. Inputs ofthe biometric information may be collected throughout an exam sessionmany times a second. The biometric information may be automaticallycollected via components of the test-taker's computer, such as, forexample, from a camera, keyboard, mouse, and/or microphone. However, inother embodiments, the biometric information may be selectivelycollected, the selection being controlled and/or determined by a proctoror administrator.

According to certain embodiments, there may be provided a system thatintegrates with online test delivery or online learning managementsystems to automatically proctor test-takers during their exams. Incertain embodiments, the system may monitor the test-taker in real-timeby collecting behavioral input from their camera, microphone, keyboard,and mouse. During the exam, a proctor or exam facilitator may observethe session in real-time and interact with the test-taker using voiceand chat. In the event the system detects aberrant behavior, the proctoror system may flag the session for further review, and send a warningsignal to the test-taker.

In certain embodiments, there may be at least four different user types.These user types may include test-takers, examadministrators/instructors, proctors/reviewers, and systemadministrators. According to certain embodiments, test-takers, using acomputing device with an Internet connection, camera, microphone and asupported operating system, may use the system to observe themselves,alongside their test-delivery or learning management system (LMS) whilethey take an exam. Prior to starting the exam, the system according tocertain embodiments may require the test-taker to submit a self-photo,show a photo ID, show their work area, and confirm exam rules. Inaddition, the system may require the test-taker to submit a biometricsample and/or answer challenge questions. After the exam has started,the system may alert the test-taker if they are performing actions thatthe system deems outside the threshold for normal.

According to certain embodiments, threshold for normal may be defined asa baseline of the environment of the test-taker established prior tobeginning the exam. The system may perform an analysis on theenvironment (e.g., light sound, and/or movement) and calibrate thesystem to that baseline. In certain embodiments, that baseline must fallinto an acceptable range prior to starting. In other embodiments, normalmay be defined by the institution or instructor. For example, if aninstructor allows for the use of an external resource such as a book,then the system may know that there might be occasional breaks in gazeand loss of face as the user references the book. According to furtherembodiments, establishing the baseline of the environment may also takeinto account initial biometric information obtained of the test-takerprior to beginning the exam.

In some cases, the system may stop the exam in the event of blatantattempt at cheating. A live proctor may also use a chat feature or voicecommunications to speak the test-taker while in the exam. For example,in the event the exam was stopped by the system, a human proctor mightbe required to review why the exam was stopped and then allow thetest-taker to continue after a brief investigation. Finally, during theexam session, the test-takers webcam video, microphone audio andcomputing device display may be recorded for later review by aproctor/reviewer or exam administrator/instructor.

In certain embodiments, exam administrators/instructors may interactwith the system through their test-delivery platform or LMS. The examadministrator/instructor may have the ability to configure some settingsof the system. These settings help define the system's rules forengagement and event tracking during a test-taker exam session. Afterthe exam session, the exam administrator/instructor can review therecorded session and review the events/flags that were created by thesystem. In addition, the exam administrator/instructor can read anynotes left by the proctor/reviewer and view some statistics about thesession in relation to other similar sessions. Lastly, the examadministrator/instructor can view reports that detail information aboutcompleted exam sessions.

According to certain embodiments, proctors/reviewers may interact withthe system through an exam scheduling system that can store and accessthe live and prerecorded test-taker exam sessions for live proctoring orpost-test session review. The proctors/reviewers can see these sessionsin a queue or list view that details if the session is being observed,or has been observed. This way, the proctors/reviewers will not attemptto observe the same sessions at a time. Next, while the proctor/revieweris observing or reviewing the session, they can see the rules for exam,view the events/session flags, view the webcam video, listen to theaudio, see the test-takers desktop and communicate with the test-takervia chat or audio in the event the session is still active. Lastly,during the live session or post exam review, the proctor/reviewer mayhave the ability to create, update, or delete session events.

System Administrators may interact with the system through theconfiguration settings for the institution through an exam schedulingsystem. The system administrator can define and update such settingssuch as application programming interface (API)/learning toolsinteroperability (LTI) keys, behavioral thresholds and sensitivities. Inaddition to these functions, the system administrator may also performall the same functions as the exam administrators and proctors.

When test-takers decide to cheat on their exams, cheating behavior maybe described by measurably consistent patterns. A system according tocertain embodiments may therefore be configured to recognize thesedistinctive patterns and automatically refine and update them usingsupervised machine learning. In certain embodiments, parts of the systemthat may be configured to recognize the distinctive patterns andautomatically refine and update them using supervised machine learningmay include a media and/or a processing server discussed herein. Assuch, certain embodiments may allow for honest test-takers to have anautomated, efficient, and convenient experience, while others may beflagged and observed live or reviewed after the exam session hascompleted. Not only does this increase the user experience, but it canalso enhance efficiency and the ability to scale various servicesprovided by certain embodiments.

Reference is made to FIG. 1 in which a system, generally indicated as10, is provided which enables remote proctoring of an exam, even inparallel with an online exam administered to a student 14. System 10includes a server 12 operatively communicating with a database 22.Server 12 communicates with students 14 at a student computer 16 throughinternet 18. Server 12 may also communicate with third party databases20 such as institutions of learning (where exams may be stored),identity bureaus (where identifying data about a student is stored), oreven provider's executable programs (such as exams or schedulingfunctions such as the virtual classroom to be discussed below) in oneembodiment of the invention.

In certain embodiments, server 12 provides an interactive web basedportal such as a web page for interacting with student 14. It should benoted that a webcam 24 is associated with computer 16.

Similarly, server 12 provides an interactive web based portal forproctors 34 and a proctor computer 30, which is also equipped with awebcam 32. Server 12 enables two-way audiovisual communication betweencomputer 30 and computer 16 utilizing webcams 32, 24. It should be notedthat computer 16, 30 may be any interactive device which allows each ofstudent 14 and proctor 34 to communicate with each other utilizing thefunctionality described below. It should be noted that the preferredembodiment is an internet-based system to facilitate the use of server12 and third party database 20 with its associated servers. However, thecomputing device may be anything compatible with a webcam or other realtime audiovisual device and capable of such communication utilizingeither the internet, radio frequency, telephone, cable TV, handheldpersonal data accessories or smart cellular phones by way ofnon-limiting example.

It should be understood as well, that all of the functionality describedbelow is capable of being performed by server 12 utilizing data storedat database 22. However, in a distributed network such as distributednetwork 10, it is contemplated that certain data and certainfunctionality may also be provided by third party 20 platforms. So bythe way of example, server 12 may create the platform for two way audiovisual communication, or provide the handshake between computer 16,computer 30 and a third party 20 platform.

Reference is now made to FIG. 7 in which a schematic diagram of theoperation of system 10 is provided by way of background. In general,server 12 facilitates the activation and use of virtual exam rooms 36.Virtual exam rooms 36 are virtual constructs, which mimic thefunctionality of a real world exam room in that it is a “place” in whichan exam may be administered under the supervision of a live proctor 34at proctor computer 30.

As will be described in greater detail below, virtual exam room 36 is aplatform, which enables the functionality, which creates an exam room inresponse to a scheduling request. In other words, virtual exam room 36is a communication between proctor computer 30 and student 16 reservedand enabled for a specific time and date and enabled upon server 12confirming the availability of a time and date for the administration ofan exam to a requesting student 14. Virtual exam room 36, as it “goeslive” also is enabled for the proctor and student webcam servers,monitoring of the student computers, presentation of the exam to thecomputers in the virtual exam room and third party authentication. Itconnects these applets in a website page.

Generally, a student 14 utilizes student computer 16 to communicate withserver 12 to access a scheduling portal 38. Scheduling portal 38provides a student with a schedule of available time periods by date andtime during which an exam is capable of being proctored utilizing thesystem of server 12. Student 14 selects an available date and time,which causes the creation of a virtual exam room 36. A proctor 34utilizing a proctor computer 30 also accesses scheduling portal 38 toidentify exam times for which a proctor 34 may be needed, or a scheduleof virtual exam rooms 36 to which proctor 34 has been assigned.

The virtual exam room 36 provides the linking platform between proctorcomputer 30 and student computer 16 as a proctor 34 and a student 16 areboth assigned to a virtual exam room 36. Utilizing virtual exam room 36and utilizing respective webcams 24, 32 and a communication platformprovided at server 12, or from a third party 20, student computer 16 andproctor computer 30 provide images respectively, in real time, of theproctor 34 and the student 16 making use of the virtual exam room 36 toadminister an exam. Virtual exam room 36 also enables proctor 34 to viewwhat is being displayed on a screen of student computer 16.

Human proctors are utilized by the invention to monitor students 14taking exams in real-time. Since the majority of all exams taken outsideof a classroom environment are delivered online, in a preferrednon-limiting example, the internet 18 is used for connection between thestudent 14 and proctor 34. The rest of the process is performed in a setof logical steps.

The Student Process

The process begins with student 14 accessing a Scheduling Portal 38, awebsite that server 12 runs utilizing database 22. A student 14 wishingto schedule a time to be proctored is provided with a login to theScheduling Portal 38 (FIG. 2). As student 14 enters log in formation,server 12 creates a unique user number that is entered into the database22 and is the identifier for all data collection related to this student14. As student 14 creates their login they are prompted by server 12 toenter contact information into system 10 including mailing address,phone numbers, and email addresses by way of non-limiting example.Student 14 is also prompted to enter their local time zone at computer16. Once the time zone is entered, all presentations of appointments andcurrent time are presented to student 14 in their indicated local time.

Once an appointment time is created by server 12 as a function ofstudent inputs at scheduling portal 38, the appointment time is storedin database 22 and recorded in Coordinated Universal Time (UTC). System10 maintains a central clock that runs on UTC and is constantly updated.

Times are converted by server 12 checking the database 22 recordassociated with the student 14 to determine what time zone they haveselected. Server 10 then retrieves the recorded difference between UTCand the student's time zone from database 22 and makes the addition orsubtraction to the UTC time of the appointment. The server 12 alsotracks dates when the difference is set to change for Daylight SavingsTime and makes the necessary addition or subtraction. This new time isthe one shown to student 14 at computer 16. All times are convertedbefore an email is sent to the student confirming information about thetime and date of an exam.

All times displayed on any web page in the schedule are converted to thestudent's time zone before being written to the page presented atcomputer 16. For example: if a student 14 makes an appointment for 11:00AM Central Time, server 12 checks the current difference between UTC andCentral. Since the difference may be −6, in this example, server 12 adds6 hours to the appointment and records it on the schedule at 5:00 PM,which is UTC time. When the student is emailed the confirmation for thisappointment, the system will check database 22 for the student's timezone. Server 12 determines that the appointment is recorded at 5:00 PMUTC and that the student's time zone is Central Time. Server 12 thenchecks database 22 for the difference between Central Time and UTC andreturns −6. Server 12 then takes 5:00 PM and subtracts 6 hours to get11:00 AM. This 11:00 AM time is then inserted into the email deliveredto computer 16.

Once a login account is created for student 14, student 14 has theability to login at a login page 200 to system 10 at any time using thecredentials they entered in the sign up process. Upon the next login,student 14 is given the option at a scheduling screen 300 (FIG. 3) toselect a date and time they wish to take an exam. Available dates 306and times 310 are shown as white boxes 302 on a calendar grid. Whiteboxes 302 can be selected at computer 16 to add an appointment at thecorresponding date 306 and time 310. Dark squares 304 indicateunavailable times.

As will be seen, system 10 utilizes live actual monitoring of studentidentification and behavior, therefore each student 14 must be assignedan available proctor 34. This schedule is a function of schoolparameters such as exam date, time and duration, by way of non-limitingexample, on the one hand, and proctor 34 availability and the maximumnumber of students that can be handled by a single proctor 34 on theother

Server 12 determines open dates by enabling an available virtual examroom 36 as a function of exam and proctoring rules created as a functionof the school parameters stored at database 22. These virtual exam rooms36 are stored in database records with associated variables held inother database records. These variables control an associated URL of thepage student 14 is to be sent to take their exam. In other words, server12 may send student 14 to a virtual room 36 created by a third party 20,or create virtual rooms itself as a function of data stored in database22. The variables may include the room's name, which is usually anumber. The name of the virtual exam room 36 is mapped in database 22 tothe login information required by the proctor 34 to connect the webcams22, 32. The number of exam rooms 36 also may correspond to rulesgoverning the capacity of exams that can be proctored at any given timeas stored in database 22 (see FIG. 10).

Each virtual exam room 36 has time slots allocated to allow an exam tobegin at staggered intervals controlled by an administrator, which iscurrently set at 15 minutes (see Admin Console Section below). Thisallows for a single proctor 34, but multiple students 34 in a virtualexam room 36. Only one exam appointment can be active in a virtual examroom 36 at a time, i.e. have a proctor 34 actively communicating with astudent 14, such as during the identification process discussed below.Virtual exam rooms 36 can be added and deleted in the Admin Console.

System 10 relies on live proctoring in a virtual exam room 36. Becauseproctoring is live, the system must be enabled to control the rate atwhich proctoring occurs. In this way, a single proctor 34 is notoverwhelmed in its attempt to initialize exams for a large number ofstudents 14 as initialization requires the steps of identityconfirmation, environment validation (e.g. no extraneous materials arepresent in a closed book exam by way of example), in addition to thevisual monitoring of the administration of the test once initialized. Tothis end, proctor 34 or an administrator of several proctors 34 isprovided with a throttle function to control the flow of students 14scheduling exams requiring proctoring.

Reference is made to FIG. 10 in which a screen shot 1000 shows a generalsettings input including an input page 1002 for inputting the manner inwhich reservations may be made such as in the instant case in which onlya single exam may be started at any one time and no more than five examsmay be proctored concurrently by any one proctor. It should be noted,that this is the gating function in the system and intentionally so. Itcontrols the number of exams, which may be given in a virtual classroom36 at the same time. Therefore, it controls the number of exams, whichmay be scheduled before a white box 302 is converted to a dark box 304at scheduling portal 38 by server 12.

The schedule also has a “throttle” for exam capacity. This functionalityallows changes to the number of virtual exam rooms 36 available forstudents 14 to reserve and will also affect the availability a student14 will see when they attempt to add an exam. This is done by theproctor's administration through an Admin Console (see Admin ConsoleSection). Operationally, human proctors can only manage starting amaximum of two exams at the same time. As discussed above, the throttlefunction allows management of how many proctors must be working at onceby controlling the number of exams that can begin at the same time andhow many exams that can run simultaneously.

For example: System 10 has ten active virtual exam rooms 36, and astudent 14 wants to schedule a two-hour exam at 10:00 AM. Server 12 willsearch for an exam room 36 that is open from 10:00 am until 12:00 pm.Once it finds a room 36, it will operate on the throttle settings storedin database 22 to determine how many exams are allowed to begin at once.Server 12 then determines whether adding the requested appointment willkeep it under that number. Server 12 then checks the throttle values todetermine the number of simultaneous exams allowed and determineswhether adding the appointment in that time frame will keep it underthat number. Once all of these conditions are met, server 12 addsstudent 14's appointment to the schedule 300.

As discussed above in connection with FIG. 7, server 12 enables aplatform for scheduling a virtual exam room 36, and as will be seenbelow, at the time and date scheduled, enables communication between acomputer 16 and computer 30 utilizing respective webcams 24, 32. Thisallows a third party 20 (which cumulatively and alternatively representsa server and database of non-proctoring entities) such as a universityto make available at its server the exam for access by computer 24. Inthis way, the proctoring provided by server 12 and the exam provided bythird party 20 run in parallel, without interference, at computer 24. Itis not necessary, unless desired by third party 20, for proctor 34 tohave access to or see the exam itself. In this way, system 10 providesan exam platform agnostic proctoring system.

Each third party institution 20 is given the opportunity to customizeits interface for their test-takers, so a student 14 will indicate theexam they intend to take by either typing the name of the exam intocomputer 16 or selecting at computer 16 from a pre-populated exam listprovided at a web page provided by a third party 20 or server 12. Thirdparty 20 can set an infinite number of variables associated with eachexam. Some of these variables can include the dates and times the examis available, the name of the exam, the associated faculty member, andexam length. These variables can be stored at database 22 eithermanually by proctor administrators through the Admin Console (See AdminConsole Section). They can also be populated automatically from theinstitution's own database 20. This is done by the institution makingthe data available in a readable format to server 12 (for example, anXML file). The data from the institution's system is requested andreturned to the server 12 any time a student wishes to add anappointment.

No matter the method that the exam variables are entered into system 10,the same process is performed by student 14 when attempting to add anappointment. Student 14 selects a date and time they wish to take anexam at page 300 of scheduling portal 36 (FIG. 3). Server 12 provides areservation page 1200 at computer 16 and server 12 populates a drop downlist 1202 of available exams with data from third party 20 or stored indatabase 22 with each entry (FIG. 12).

Student 14 selects any variables that the institution desires at a page400 (FIG. 4). When the student 14 selects the exam and attempts to addit to the schedule 300, server 12 will first determine the associateddatabase entries to determine whether the exam is available at the dateand time the student 14 is requesting. If it is not, server 12 sendsstudent 14 an error message at computer 16 and is advised by server 12as to when an appointment is available. If the exam is available in thetime frame student 14 is requesting, system 10 performs its normal setof checks for available virtual exam rooms 36 as a function of rulesincluding throttle settings. If server 12 determines that settings allowit, an appointment for student 14 is added to the schedule 300.

Student 14 is given the opportunity to print a confirmation page andserver 12 sends a confirmation via email to computer 14. Student 14 isthen returned to the calendar grid 300 of scheduling portal 36.

Server 12 generates a countdown clock by comparing current time to thescheduled appointment time, which is displayed as a clock 308 in theschedule 300. Clock 308 counts down until the time the exam is scheduledto begin. This countdown clock is calculated by server 12 firstrecognizing that the student 14 has an appointment on the schedule 300.Server 12 then takes the time of the next appointment student 14 has onschedule 300 (in UTC) as stored in database 22 and subtracts the currentdate and time (in UTC) from the appointment date. The difference is thendisplayed on the schedule grid page 300 as clock 308 and is periodicallyupdated; every second in a preferred embodiment (FIG. 6). Once the examis scheduled to begin, the countdown clock 308 becomes a button 312 thestudent can select to begin the proctoring session (FIG. 6). Thisprocess is done when server 12 calculates the difference between theappointment time and current time is zero or a negative number. Thebutton, in a non-limiting embodiment, is an html link to a URL thatcorresponds to the virtual exam room 36 in which the student'sappointment is scheduled.

Once the student selects virtual button 312, server 12 provides a newpage at computer 16 to begin the proctoring process. The first step isto connect proctor 34 and student 16 via a live webcam session. The newweb page automatically detects the student's webcam 24 and connects themto a live proctor's 34 webcam feed at computer 30 and webcam 32 (FIG.8). This is done via a live video conference executable application intothe page. This process may be done in a preferred embodiment through aplatform provided by third party 20, but can also be done by server 12.The webcam connection allows the student 14 and proctor 30 to see andhear each other. The proctor 30 then greets the student 14 and talksthem through the next steps.

Student 14 is then instructed to proceed to the next step utilizing aninterface 800 (FIG. 8, step two). Student 14 initializes a remoteconnection to proctor 34, which allows proctor 34 to monitor thestudent's screen and system for computer 16. The connection also allowsthe proctor to control the peripherals (mouse and keyboard) of computer16 and monitor the system health of computer 16, which aids in technicalsupport needs. This is done through a third party 20, but can be done byserver 12. This method currently works on both Windows and Maccomputers. This capability is tied to the technology used, whether froma third party or from server 12.

From there, the student 14 is instructed to prove their identity throughan approved authentication method. Examples of authentication methodsinclude: Proctor 34 instructs the student to present pictureidentification to webcam 24. The type of identification acceptable isdetermined by the institution for which the student 14 is taking theexam and stored either at database 22 or third party database 20 andsent to proctor 34 as part of the process. Student 14 may also be askedto enter their name and home mailing address into a pop up windowpresented at computer 16, but monitored at server 12 and/or computer 30.This data may be transmitted to a third party 20, which in responsereturns a series of questions generated from data stored at third partydatabase 20 related to the student 14 at computer 16. This could bepublic and privately purchased data that deals with previous addresses,real estate history, old phone numbers, and other types of informationgenerally known collectively only to the student 14 by way ofnon-limiting example. Student 14 is to answer the questions whileproctor 34 watches via webcam 24 and remote connection. If student 14scores a high enough percentage on the authentication exam, they areallowed to proceed.

The percentage can be determined by the institution or proctor 34. Thepassing or failing of this authentication exam, as well as the contentsof the questions are stored in the database of third party 20. Thestudent may also have a picture taken of them through webcam 24 by theproctor 34. This picture is then stored in database 22 and associatedwith student 14. It can be viewed by proctor 34 at any time to verifythat the appearance of the student 14 who has connected via webcam 24appears to be the same student 14 who has taken previous exams. Anycombination or all of these three methods can be used to authenticate astudent's identity at the discretion of the institution.

After the completion of the steps above, student 14 is instructed tobegin the exam. If the exam is being delivered online, student 14 isdirected to open another browser window and navigate to the exam on thewebsite of third party institution 20. System 10 does not require theexam to be delivered in any specific fashion and works with any LMS. Thereason the system works with any LMS is that system 10 enables viewingat computer 30 the images displayed on computer 16 so that proctor 34 issimply monitoring what is being shown on the student's monitor. Forexample, if an institution wants the student to use a Microsoft Wordduring the exam, the proctor simply monitors the student doing so andmakes sure they use nothing else. This allows system 10 to be platformagnostic. The proctoring is going on in parallel with whatever elsestudent 14 is doing on computer 26.

If the exam is being delivered via paper, proctor 34 will instruct thestudent to complete the exam while on webcam 24 and will make surestudent 14 completes it in the manner directed by the institution.

After the exam is complete, student 14 “clocks out” by entering an endinstructed at computer 16 and proctor administrators, as well asselected administrators of the institution, are able to generateActivity Reports 900 (not shown) detailing all past proctoring sessions.If the exam times out then proctor 34 can also end the exam fromcomputer 30, or server 12 may discontinue access to the exam. Anyvariable that the institution wishes to be tracked can be displayed fromthis report. The report is generated by providing to computer 30 anumber of drop-down menus 1302 at a page 1300 that contain variablesthat allow the Administrator to narrow down the data that is displayedin any given report (FIG. 13). The number of drop-down menus can becustomized at the desire of the institution. A report 902 can contain,but is not limited to, pictures 904 of the student 14 who appeared onthe webcam (if it was recorded), the start 906 and end 908 times of theexam, the time the student began and ended the exam, and notes 910 fromthe proctor (FIG. 9) shown at screen 900. Administrators are also ableto see future exam appointments, manage lists of exams, and monitordaily schedules.

The Proctor Process

Operation of system 10 will now be explained as utilized by proctor 34.Proctors 34 are provided a different type of login into the System 10.These system 10 logins allow the proctors 34 to access the schedule 300of scheduling portal 38 in a view that allows proctors 34 to quickly seethe information at computer 30 they need in order to proctorefficiently.

Proctors 34 login from computer 30 though login process similar inoperation to that used by a student 14 (FIG. 2). Once proctor 34 logsin, server 12 provides access to a page 1100 at computer 30 thatdisplays the schedule 1102 in “Proctor View.” Schedule 1102 is displayedshowing the total number of virtual exam rooms 36 and all potentialstart times 1104 for the current day (FIG. 11). All proctor 34appointments 1106 for that day are displayed in line with theircorresponding virtual exam rooms 36 and the time they are set to begin.All appointments 1106 are color-coded by the institution for which theexam is being proctored. These colors are set in the Admin Console (seeAdmin Console section). This allows the proctor 34, at a glance, to knowwhat login information to use to connect the webcams and what school'sguidelines are to be consulted for this appointment 1106 as stored indatabase 22.

Above the display of virtual exam rooms 36 is a calendar 1100 accessiblefrom computer 30. Each day on calendar 1108 can be accessed to displayall appointments for that day in the corresponding virtual exam rooms36.

Any appointment on the schedule can be clicked by a proctor 34. Thiswill provide an accessible menu on page 1100 that offers proctor 34 thefollowing options: Edit; Notes; Contact; and Cancel. (FIG. 14).

The Edit Function:

When the proctor 34 selects Edit button 112 at computer 30, server 12connects proctor 34 to a new page 1500 that pulls all data from thedatabase 22 or third party 20 associated with this appointment 1106(FIG. 15). Proctor 34 can edit any variable associated with thisappointment 34. This can include, by way of example, but is not limitedto the following: The ability to change the student 14 listed with thereservation 1106; The ability to change the name of the exam; Theability to change the start and end times of the appointment 1106; Theability to change the notes the student may have left at the time theappointment was made.

The Notes Function:

By selecting Notes button 1114 server 12 provides access to a windowthat allows a proctor 34 to add text to database 22 and associate thistext as “notes” about the proctoring appointment 1106. The notes mayincluded, by way of example, but are not limited to, something student14 did during appointment 1106, something proctor 34 noticed, ortechnical issues proctor 34 had connecting to the student 14. This datais displayed in the Activity Report 900 and can be viewed by proctors 34and administrators from the institution.

Contact Function:

By selecting Contact button 1116, server 12 provides a pop up window1600 that contains the contact information 1602 contained in database 22that corresponds to this student's account (FIG. 16). Pop up window 1600may also display the picture 1604 on file for that student 14, allowingthe proctor to verify visually the student's identity as apart of anauthentication process.

Cancel Function:

Selection of Cancel button 1118 causes server 12 to allow proctor 34 tocancel an appointment 1106. A reason for cancellation may be included ina text box that will be recorded in database 22 and linked to thisappointment 1106. This information can be pulled from the “CancellationReport” in the Admin Console (see Admin Console).

Proctors also have access to the Admin Console, which provides certaindegrees of initialization and administrative control over system 10 andwill be described below. Proctors 34 and Administrators have access to aScheduling Portal's Admin Console 1700 (FIG. 17). In the Admin Console1700, variables contained in database 22 that control mainline settingsof the Schedule Portal 36 can be edited.

The Schedule Button:

Selecting Schedule button 1702 from Portal's Admin Console 1700 causesserver 12 to return the page displayed at a computer to the “ProctorView” of the schedule 1102.

The Users Button:

By selecting a Users button 1704, server 12 enables an Administrator toadd, edit, and delete information about all proctors 34, administrators,and students 14. This is done through a menu 1800 that allows displaysof all profile information for each account (FIG. 18). If theAdministrator wishes to add a new user, they enter the information froma remote computer corresponding to the new user and select the “add”button 1802 Server 12 causes a new set of records to be added to thedatabase 22 and the user is given a unique User Number. If theAdministrator wishes to edit an existing user, they select the user'sname from the drop down box 1804 labeled “User.” All existing profiledata in database 22 for the user is displayed and can be edited by theproctor 34. When finished, selecting the “Update” button (not shown)will cause server 12 to update all the information associated with thisuser in database 22. Selecting a “Delete” button will cause server 12 todelete all information about the user from the database 22.

The Test Station Button:

Selecting a Test Station button 1706 causes server 12 to enable theAdministrator to create and edit all of the virtual exam rooms 36. Adrop down box appearing at their computer enables administrators toselect an existing exam room 36 or add a new exam room 36. All of thevariables mentioned in the virtual exam room section above can be editedhere. Virtual exam rooms 36 can be deleted from the database 22 (orthird party 20) by selecting the exam room 36 in the drop-down menu andselecting the “delete” button.

The General Button:

Selecting General button 1708 causes server 12 to enable anadministrator to edit many of the general settings of the site such asthe throttle rate at page 1000 (FIG. 10). In addition to thesefunctions, the main contact information for the site is edited here.Administrators can also use this section to edit the hours of availabletime slots by selecting an “open and close” time. This changes thedisplayed time slots that are shown to proctors 34 and students 14 inthe schedule grid 300. The increment at which time slots are available(currently set at 15 minutes) can be changed here as well. The Terms ofService for the site can be edited here as well.

The User Types Button:

Selection of a User Type button 1710 causes server 12 to enable theAdministrators to edit the data stored in database 22 for differenttypes of users in the system, such as “proctor,” “student,” and“institutional admin.” In editing these user types, variables thatcontrol their access to certain sections of the schedule 300 and AdminConsole 17 and permissions to perform certain access can be edited here.Some of these variables include, but are not limited to, the ability toadd appointments, the ability to view proctor comments, and how long auser is allowed to stay logged into the system without being prompted tore-enter their login credentials.

The Schools Button:

Selection of a Schools button 1712 causes server 12 to enable anAdministrator to add, delete, and edit data about an institution storedin database 22 (FIG. 19). Server 12 provides access to a page 1900 wheresettings for institutions can be customized. The sections include, butare not limited to, setting up the school to use exam lists (specificsettings for each list is controlled as described in the Exam Listsection); and editing the email message that is sent to a student 14 ofthe institution when they make an appointment.

The Exam Lists Button:

Once an institution has been edited utilizing page 1900, theAdministrator selects the Exam Lists button 1714. Server 12 in responseto selection of the Exam List button 1714 enables an Administrator toadd exams to database 22 or edit exam information in database 22. Asdiscussed above, each exam from each institution and variablesassociated with it are stored and mapped in database 22. These variablesmay be set, edited or deleted as discussed above in the Exam List andAppointment Section.

The Cancel Codes Button:

Selecting a Cancel Codes button 1718 causes server 12 to enable theAdministrators to set generic reasons that proctors 34 can select whencanceling a student's exam appointment.

The History Button:

Selecting a History button 1720 causes server 12 to enable anAdministrator to access reports from database 22 on the addition ofusers, the addition or changing of appointments, and the changing ofsystem settings as stored in database 22 as server 12 performs thefunctions discussed above. All changes in database 22 regarding thesefunctions are logged in another section of the database, allowing forchange reports to be displayed. This information is used fortroubleshooting a problem after it appears to have happened to pinpointthe changes that may have caused it.

The Activity Report Button:

Selecting an Activity Report button 1722 causes server 12 to provideAdministrators with access to the same Activity Report 900 thatinstitutional administrators use.

Cancellation Report Button:

Selecting a Cancel Report button 1724 causes server 12 to provideAdministrators with access to report on all appointments that have beendeleted or canceled. In the same manner that Activity Report 900 isgenerated, relevant variables can be selected at a page 2100 to narrowthe data returned so that all that is used is the information desired(FIG. 21).

Email Button:

Selecting Email button 1726 causes server 12 to enable Administrators tosend an email to any or all users of Schedule Portal 38 at the emailaddress recorded in the database 22 that corresponds to their userprofile.

The innovation behind the above approach is the virtualization of thetraditional classroom testing experience. The use of human interactioncombined with authentication methods maximizes the security andintegrity of the proctoring session without making the process difficultfor the student. Other systems focus on automation and not monitor inreal time.

FIG. 22 illustrates a system according to certain embodiments. Thesystem may include a scheduling system 3100. Such a scheduling systemmay be used to create a proctoring platform. Although FIG. 22illustrates a scheduling system, other technologies may be used tocreate the proctoring platform in other embodiments. The schedulingsystem 3100 may be responsible for storing and processing data thatpertains to the sessions. The scheduling system 3100 may also be theideal interface used to aggregate, process, and display data for allusers, which may include test-takers, proctors, and institutions. Thedata may include, but not limited to appointment date/time, userinformation, session rules, session events, session timeline, sessionrecordings, session authentication, session notes, session photos, examinformation, test taker device input, and proctor input. Further, thescheduling system 3100 may be based on a model, view, controllerarchitecture, and may support an application programming interface thatis used to transfer data between different components. In addition, thescheduling system 3100 may configure all aspects of the session.

FIG. 22 also illustrates that the system may include a media servercomponent 3200. The media server component 3200 may be responsible forrecording, streaming, and processing audio and video. Further, audioand/or visual data may be streamed to the media server component 3200.In addition, the media server component 3200 may analyze and parse thestream for various data points such as, but not limited to: facedetection; face profile detection; lighting levels; audio levels; motionlevels; and object detection.

FIG. 22 further illustrates that the system may include an end-usersoftware component 3300. The end-user software component 3300 mayreceive most of the interaction. In addition, the end-user component3300 may be responsible for transmitting and displaying various types ofinformation including, but not limited to: webcam audio and/or video;desktop streaming; device interface events; system information; Internetconnectivity; system state; user information; appointment information;session events; session state; and voice and/or chat communications.

FIG. 23 illustrates a user flow and interaction model according tocertain embodiments. Step 1 includes a user interacting with theend-user software to login to the system. At Step 2, the schedulingsystem may interact with the end-user software to determine the user'srole. Further, Step 3 shows that after the login, user roles have beendetermined, and if a student would connect to the media server and couldbe observed in real-time by a remote proctor. Further, Step 4 shows thata proctor could interact with the media server through the schedulingsystem to interact with the student or test-taker in real-time.

FIG. 24 illustrates partial test-taker user flow according to certainembodiments. In particular, FIG. 24 illustrates a user flow for atest-taker who may or may not have an appointment. The test-taker maylog into the system, which may be the scheduling platform that has notbeen enhanced with automated proctoring, and log into their testdelivery platform. The end-user component may establish, with thescheduling component that a fulfillment or session exists or does notexist. If one exists, the user may proceed to the system pre-checks. Incertain embodiments, the system pre-checks may include, but not limitedto, at least the following: checking the users' environment (e.g.,lighting, sounds, and/or movement); validating that their computer has afunctioning camera, microphone, available bandwidth, and open networkcommunication; and validating their identities. If one does not proceedand there are no errors, then a session may be created, and the user maythen proceed to the system-pre-checks.

As illustrated in FIG. 24, at 4100, the student may start the processfor initiating a session. At 4150, the system may attempt to figure outwho the user is. In addition, at 4200, the system may either sign theuser, ask the user to sign-in, or require the user to sign-up. At 4250,if the user is authorized at 4200, the system may check for an existingreservation/fulfillment, or create a new one. Further, at 4300, if theuser is not authorized at 4200, the system may alert the user to sign-inor sign up. At 4350, the system may check for a positive response toproceed to start the exam. At 4400, if the system received a negativeresponse at 4350, the system may either direct the user to validate theerror, or direct the user back to the exam page.

As further illustrated in FIG. 24, at 4450, the system may start theexam check-in process. In addition, at 4500, the system may determinethe type of error message and either try to create areservation/fulfillment again, or direct the user back to the exam page.Further, at 4550, it is shown that the user is/was redirected back tothe exam start page. Finally, at 4600, the system may try to createreservation/fulfillment. Further, at 4600, success may start the examcheck-in process and failure may lead the student back to the exam startstate.

FIG. 25 illustrates a proctor user flow according to certainembodiments. For instance, FIG. 25 illustrates an overview of a proctorflow. According to certain embodiments, the user may be able to selecttheir view mode 5100 and 5650. The view mode may include an ALL mode5150, which shows all sessions in a given time window. The view mode mayalso include an automated mode 5200, which only shows automated sessionsin a given time window. Further, the view mode may include a Live mode5250, which only shows Live sessions in a given time window. Inaddition, certain embodiments may include a Dashboard that shows variousstatistics such as the current reservations per hour 5350, and have theability to search for any session from this page.

According to certain embodiments, the session list 5400 illustrated inFIG. 25 may show four different functions. The first function may be theproctor 5450. Clicking this may bring up the test-takers media stream.In this view, the proctor may have access to a predefined set of examinstructions that enable the proctor to perform their function, desktopview, timeline events, authentication results, and any informationnecessary proctor or review the session. The proctor may also have theability to communicate with the test-taker in the event it is a livesession.

The second function may be a view function 5500. If the session iscomplete, this may be similar to the proctor function. Further, thethird function may be a reschedule function 5550. If this is a livesession, the user may have the ability to reschedule the session.Finally, the fourth function may be a cancel function 5600. If this is alive session, the user may have the ability to cancel the session. Incertain embodiments, the manager may have the ability to assign sessionsfor proctoring or review. In other embodiments, proctors may only seesessions that they should be responsible for or assigned. The interfacemay indicate if a session needs attention.

FIGS. 26 and 27 illustrate a test-taker user flow according to certainembodiments. The test-taker user experience may be composed of thefollowing states: schedule test; scheduled test; pre-test; test; andtest end. In the scheduled test state 6200, the test-taker may navigateto their test location in the LMS 6220, and have the ability to scheduletheir exam 6205 or take it on-demand, if applicable. When scheduling theexam, the test-taker may select the appropriate date/time and then payfor the session 6210 if applicable. If no scheduling or payment isinvolved, then the test-taker may be taken to the pre-test state 6135.If scheduling and/or payment were involved, the test-taker may be in thescheduled state 6225.

According to certain embodiments, in the scheduled state 6225, thetest-taker may have the ability to reschedule 6230 or cancel 6235 theirappointment. If the test-taker decides to reschedule or cancel, thetest-taker may go back to the schedule test state 6200. In this state,the test-taker may see a countdown timer with the time until theappointment along with a link to test their equipment. Further, in thepre-test state 6135, certain embodiments may engage security features6140 in the end-user software based on institution-defined settings,being audio/video/desktop recording/streaming 6145, 6150 and prompt examrules 6155. As illustrated in FIG. 26, the audio/video stream 6145 anddesktop stream 6150 may be in communication with the media server 6130,and the media server 6130 may be in communication with the ArchimedesAPI 6125. In certain embodiments, the Archimedes API 6125 may be anapplication programming interface for the scheduling system that allowsfor external communications between applications or servers.

According to certain embodiments, the security features 6140 mayinclude, but are not limited to the following: additional payments;taking the test-taker's photo; taking a photo of the test-taker's ID andchecking it's authenticity; recording a room pan; taking a biometricsample (e.g., audio, keystroke, etc.); system process checks; requireone screen or full screen; disable additional tabs, printing, clipboard,and right-click; challenge questions; audio floor analysis; and facialcheck and/or tracking.

Once the test-taker has completed the automated launch process, a buttonmay be available to start the exam 6160. After exam launch at 6160, thetest-taker may be directed to the Learning Management System (LMS) 6175(i.e., Instructure Canvas, Blackboard, Moodle). The purpose of the LMS6175, according to certain embodiments, may be to house, organize,communicate, and deliver learning and assessment materials for thestudent. According to certain embodiments, the end-user software mayrelease the exam password, and the user may be in the test state 6180.At anytime during the pre-test state 6135, the test-taker may connect toa proctor via a chat 6170, get assistance, or ask questions. In certainembodiments, in the test state 6180, the end-user software may haveunlocked the exam 6185, and will continue to stream audio, video, and/ordesktop media, and usage data to the media server 6130 and schedulingsystem server for real-time analysis. In addition, at 6185 under thetest state, the test-taker may be continually monitored. According tocertain embodiments, 6185 may represent the collection of the input bythe proctoring software that is downloaded on the test-taker's machine.The information may be sent to the server, which may then be provided tothe media server 6130 and Archimedes API 6125, and then may further beprocessed by a processing server. The information may then be used bythe Archimedes API 6125 and/or the processing server to determine if thetest-taker is engaging in questionable behavior such as cheating. Someof the collected input may include, but not limited to the following:interval checks of processes, tabs, window focus, and resize; mouseclicks and keystrokes; time calculations; lighting levels; audio levels;and movement tracking.

According to certain embodiments, in the event of a blatant tamperingwith the end-user software is detected, the exam may be forced to submitan incident report. In certain embodiments, the Archimedes API 6125,media server, and/or the processing server may determine if there hasbeen a blatant tampering with the end-user software and submit anincident report, and such determination may be made automatically.According to other embodiments, a proctor of administrator can also makethis determination and submit an incident report manually. If there areno incidents, the exam may be submitted, and the end-user software maybe placed in the end state 6195. According to certain embodiments, inthe end state 6195, the exam has been completed (natural or forced), theend-user software may cease all audio, video, desktop, and/or datastreaming, and return to the schedule state 6225.

Before entering the above-described test states, a test-taker may firstbe prompted to download and install the end-user software prior totaking an exam. Once the end-user software has been downloaded, thetest-taker may login 6100 with a direct scheduling system account orthrough a single sign-on option from the test-taker's Test DeliveryPlatform or LMS.

As illustrated in FIGS. 26 and 27, the test-taker may login with a useraccount or Ucard. If the test-taker did not have an account, they may beprompted to create an automated identity check and sign-up as a new user6105 and create an identification card such as a Ucard 6110. During theprocedure of creating an automated Ucard, the test-taker's equipment maybe checked. Further, recording may be started, and the test-taker may beprompted to complete an account profile. The test-taker may also beprompted to show his/her ID, and optionally answer challenge questionsand complete a keystroke security check. Upon completion of creating theUcard, the test-taker may automatically be assigned a “student” userlevel, and at this point may be ready to schedule or take exams.

At 6115, the test-taker may be detected to determine a user type 6115.After the user type is detected, the test-taker may be presented withvarious navigation options 6120. For example, the navigation options6120 may include: view/open schedule that includes single sign-on intothe scheduling system to see scheduled exams, and run reports;help/support to provide access to the FAQ and Live Chat Services;un-install to remove the software; and disable to disable the software.

After the automated identity check, the test-taker may automatically beenrolled as a “test-taker” user and would be ready to schedule or takeavailable exams. According to certain embodiments, after a successfullogin, the test-taker may have the ability to view and/or open scheduleand exam, get help/support (provide access to frequently asked questionsand chat services), and uninstall the software and/or remove thesoftware.

FIG. 28 illustrates an instructor or exam administrator user workflowaccording to certain embodiments. Here, the user may interact with theirLMS, test-delivery platform, help/support through the end-user software.The end-user software may interact with the scheduling system, mediaservers, and the LMS and/or test delivery platform. The instructor orexam administrator user experience may be composed of the following:installation; login; navigation; create/read/update/delete (CRUD) test;ready to test; and test complete.

According to certain embodiments, the user may be invited to downloadthe end-user software via email notification or by some other means. Theuser may also navigate to a website to download the software. Once theend-user software has been installed, the user may login 7100 through aschedule system account or via a single sign-on from their LMS or testdelivery system. Following the login, the user may proceed through steps7105, 7110, 7115, and 7120, which are similar to those of 6105, 6110,6115, and 6120 described with respect to FIGS. 26 and 27. If the userdid not have an account, they would be prompted to create an automatedidentity verification, and create an automated ID card, such as a Ucard.During the procedure of creating an automated Ucard, the instructor'sequipment may be checked. Further, recording may be started, and theinstructor may be prompted to complete an account profile. Theinstructor may also be prompted to show his/her ID, and optionallyanswer challenge questions and complete a keystroke security check

The instructor may then be automatically assigned an “institution” userlevel if using the Learning Tools Interoperability (LTI) user-rolematching. In the event the user is not using LTI user-role matchingduring the account creation process, they will designate if they are atest-taker or an institutional user. According to certain embodiments,the LTI may be a data communication specification. This specificationmay be used by LMS creators and organize their user groups into specificroles, such as administrator, faculty, instructor, learner, etc. Thisfeature, according to certain embodiments, may allow for the ability tocomply with the standard and accept that user role in the schedulingsystem.

Upon a successful login, the user may have the following navigation 7120options from the end-user software: view/open schedule that includessingle sign-on into the scheduling system to see scheduled exams, andrun reports; help/support to provide access to the FAQ and Live ChatServices; un-install to remove the software; and disable to disable thesoftware.

During implementation of the CRUD test 7130, the test taker may have theability to create/read/update/delete their test from their LMS interface7135. Further, the user may have the ability to designate if the testwill be live proctored or automated 7140. Depending on which settingsthey select, they may be provided with the appropriate form fields tosetup their test such as, for example, an automated settings 7145 formfield or a live settings 7150 form field. At this point, the test-takersmay take or schedule their test; the settings could be viewed, updated,or removed.

At the ready to test state 7155, the test may be ready to be taken.Depending on what settings were selected, the user may receive thefollowing notifications 7160 via an indicator, a short message service(SMS), or email: scheduled; rescheduled; cancelled; testing now; testsubmitted; or incident report. Once a test-taker has submitted a test,the instructor/institution may review 7165 the session recordings. Thesessions may contain proctor notes and/or automated flags. Further, theuser may view a session in progress. If there is a session of particularinterest, the user may create a report 7170 from it or export someinformation. Once all the test-takers have submitted their exams, theinstitutional user may continue to review sessions or create reports foras long as the institution's data retention settings allow 7175.

FIGS. 29-32 illustrate infrastructure architectures according to certainembodiments. In particular, FIG. 29 illustrates an overview of certainembodiments of the invention with the added description of the LMS toshow how certain embodiments of the invention would interact with it.Further, FIG. 30 illustrates a way that FIG. 29 could be implementingusing native .webm AV streaming, according to certain embodiments. Inaddition, FIG. 31 illustrates a second way that FIG. 29 may beimplemented using different formats of AV streaming, according tocertain embodiments. Further, FIG. 32 illustrates a third way that FIG.29 may be implemented using a third party PAAS tool for AV streaming,according to certain embodiments. The potential infrastructurearchitectures may be considered to implement certain embodiments of theinvention.

As illustrated in FIGS. 29-32, the test takers 8000 may represent thestudents or users of embodiments of the invention. Further, the browserextension 8005 may be an embodiment of the client facing part of certainembodiments of the invention. In addition, the Learning ManagementSystem (LMS) 8010, as noted above, may be an instructure canvas,blackboard, and/or Moodle. Further, the media/turn/stun WebRTC server(s)8015 may be a method of signaling WebRTC communications, and theProctorU Scheduling System 8020 may be an embodiment of the backendprocessing of certain embodiments of the invention. Further, theproctors 8025 may be observers of the exam sessions and users of certainembodiments of the invention.

As further illustrated in FIGS. 29-32, the Amazon S3 may represent amethod of storing data according to certain embodiments. In addition,the processing server(s) 8035 may represent a method of processingaudio, video, and/or behavioral information according to certainembodiments. In addition, the scheduling system DB or separate DB 8040may represent an approach of housing the data for the automated sessionsoutside of the ProctorU Scheduling System 8020. Further, the takbox PAAS8045 may represent an approach of using an existing technology serviceto assist with part of certain embodiments of the invention as itpertains to audio or video communication, recording, and transfer. Inaddition, the compression/conversion server(s) 8050 may represent anapproach in the data flow to convert the audio, video, or behavioralinformation into an acceptable format for more efficient data processingaccording to certain embodiments.

FIG. 33 illustrates a supervised machine learning flow according tocertain embodiments. The supervised machine learning may be implementedon the media server or the processing server, and may be controlled bycertain embodiments of the invention. In particular, FIG. 33 illustrateshow audio/visual/device input patterns may be stored and analyzed usingsupervised machine learning. With the assistance of proctor input fromobserving sessions, certain embodiments may establish the differencebetween normal and aberrant testing behaviors. For instance, accordingto certain embodiments, patterns of cheating may be captured, received,and/or obtained by the system implementing the supervised machinelearning. The patterns, in certain embodiments, may be monitoredconstantly, along with additional ID validations of the test-taker, forsigns of cheating via the machine learning. Alternatively, in otherembodiments, such input of information regarding the patterns ofcheating may be provided directly as input to the system implementingthe supervised machine learning. Such patterns may be based on variousfactors of the test taker, including, for example, external resources.In certain embodiments, the external resources may include, for example,the test taker leaving the view of the camera for a certain amount oftime such as for about 15 seconds. Other lengths of time of the testtaker's lost face may also be monitored. The lost face may be followedby an answer submission when the exam is no resources allowed. Further,the system may look for cheating behavior of the test-taker to occur atleast 3 times. However, in other embodiments, the number of times thatcheating behavior is detected can be adjusted to a desired amount.

According to certain embodiments, each time a proctoring session occurs,the session patterns may be compared to the initial patterns. Theinitial patterns may be pre-programmed behavioral patterns that aredesignated to be representative of cheating patterns. In certainembodiments, when cheating occurs during an exam session, the actualbehavior patterns of the test-taker may be compared to the originalpre-programmed patterns. Thus, certain embodiments may also make adetermination if the observed patterns match those of the pre-programmedpatterns. If the patterns match, then the system may mark the sessionfor cheating. In addition, according to certain embodiments, it may bepossible for the comparisons to shape the initial pre-programmedpatterns into something more accurate via machine learning. That is,according to certain embodiments, recognition of distinctive patternsmay be continually refined and updated. Further, in other embodiments, ahuman proctor may then review the session and either support theassertion of cheating or deny it. In either case, the human assertionmay strengthen the overall behavior of certain embodiments of theinvention.

FIG. 34 illustrates an example of a system according to certainembodiments. It should be understood that each block of FIGS. 1-33 and35, or any combination thereof, may be implemented by various means ortheir combinations, such as hardware, software, firmware, one or moreprocessors and/or circuitry. In one embodiment, a system may includemultiple devices, such as, for example, a computer 610 and a server 620.The system may include one or more computers 610 and more than oneserver 620, although only one computer 610 and one server 620 are shownfor the purposes of illustration.

The computer 610 may be any computing device or a user computerassociated with students 14 or proctors 34 that can take the form of atraditional server/desktop/laptop; mobile device such as a smartphone ortablet; etc. Computing devices may typically include one or moreprocessors coupled to data storage for computer program modules anddata. Key technologies may include, but not limited to, themulti-industry standards of Microsoft and Linux/Unix based OperationSystems; databases such as SQL Server, Oracle, NOSQL, and DB2; Businessanalytic/Intelligence tools such as SPSS, Cognos, SAS, etc.; developmenttools such as Java, NET framework (VB.NET, ASP.NET, AJAX.NET, etc.); andother e-Commerce products, computer languages, and development tools.Such program modules may generally include computer program instructionssuch as routines, programs, objects, components, etc., for execution bythe one or more processors to perform particular tasks, utilize data,data structures, and/or implement particular abstract data types. Whilethe systems, methods, and apparatus are described in the foregoingcontext, acts and operations described hereinafter may also beimplemented in hardware.

In certain embodiments, the server 620 may be a node, host, or server ina network. For example, the server 620 may be a remote desktopviewing/control server, an audio/video server, a client server, anadministration server, a database server, or other similar type servers.

Each of these devices may include at least one processor respectivelyindicated as 614 and 624 for processing information and executinginstructions or operation. Processors 614 and 624 may be any type ofgeneral or specific purpose processors. While a single processor 614 and624 is shown for each device in FIG. 34, multiple processors may beutilized in each device according to other embodiments. In fact,processors 614 and 624 may include one or more general-purposecomputers, special purpose computers, microprocessors, centralprocessing units (CPU), digital signal processors (DSPs),field-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), and processors based on a multi-core processorarchitecture, as examples, or other comparable devices. The processors614 and 624 can be implemented as a single controller, or a plurality ofcontrollers or processors.

At least one memory (internal or external) can be provided in eachdevice, and indicated as 615 and 625, respectively. The memory mayinclude computer program instructions or computer code containedtherein. The processors 614 and 624 and memories 615 and 625, or asubset thereof, can be configured to provide means corresponding to thevarious blocks and processes shown in FIGS. 1-33 and 35.

Memories 615 and 625 can independently be any suitable storage device,such as a non-transitory computer-readable medium. A hard disk drive(HDD), random access memory (RAM), flash memory, or other suitablememory can be used. The memories can be combined on a single integratedcircuit as the processor, or may be separate from the one or moreprocessors. Furthermore, the computer program instructions stored in thememory and which may be processed by the processors can be any suitableform of computer program code, for example, a compiled or interpretedcomputer program written in any suitable programming language.

The memory and the computer instructions can be configured, with theprocessor for each device, to cause a hardware apparatus such asapparatus 610 and server 620 to perform any of the processes describedherein (see, for example, FIGS. 1-33 and 35). Therefore, in certainembodiments, a non-transitory computer-readable medium can be encodedwith computer instructions that, when executed in hardware, perform aprocess such as any one of the processes described herein. Accordingly,in certain embodiments, a non-transitory computer-readable medium can beencoded with computer instructions that, when executed in hardware,perform any of the processes described herein. Alternatively, certainembodiments of the invention can be performed entirely in hardware.

FIG. 35 illustrates an exemplary flow diagram of a method according tocertain embodiments. According to certain embodiments, the methodillustrated in FIG. 35 may be performed by a computer, such as acomputer of the embodiments described above. At 105, the method mayinclude checking a test-taker's computing device for compatibility, andfor content that provide unauthorized aid to the test-taker during atest session. The method may also include, at 110, taking a photo of thetest taker. In addition, at 115, the method may include recording thetest taker by performing a room pan while analyzing the surrounding ofthe test-taker for unauthorized objects. At 120, the method may includevalidating the identity of the test-taker by way of at least one ofpresenting challenge questions, obtaining voice biometrics, or obtainingkeystroke biometrics. Further, at 125, the method may include recordingthe test-taker's audio or video and desktop feed.

FIG. 35 also illustrates at 130, determining if the test-taker isexhibiting questionable behavior. According to certain embodiments, thequestionable behavior may be determined based on an analysis of thecompatibility of the test-taker's computing device, the test-taker'sphoto, the room pan, validation of the test-taker's identity, andrecordation of the test-taker's audio or video and desktop feed. At 135,the method may include suspending the testing session if it isdetermined that the test-taker is exhibiting questionable behavior.

At 140, the method may include providing the test-taker with test rulesdefined by an institution of the test taker. Further, at 145, the methodmay include checking the test-taker's computing device for applications,code, or materials used to provide unauthorized aid during the testingsession. At 150, the method may include sending the test-taker a warningsignal when it is determined that the test-taker is exhibitingquestionable behavior. In addition, at 155, the method may includecommunicating with a live proctor when it is determined that thetest-taker is exhibiting questionable behavior. Further, the method mayinclude at 160, adjusting system settings for proctored test sessions,at 165, reviewing completed proctoring test sessions, and at 170,viewing sessions flagged for questionable behavior of the test-taker.

According to certain embodiments, it may be possible to provide a remoteonline proctoring system that is capable of describing cheating behaviorby measurably consistent patterns. It is also possible to provide asystem that may be configured to recognize these distinctive patternsand continue to automatically refine and update them using supervisedmachine learning. Such capabilities may allow for increased userexperience, and enhancement in efficiency and ability to scale remoteonline proctoring services.

According to other embodiments, it may be possible to solve the variousproblems exhibited by conventional online remote proctoring discussedabove. For instance, according to certain embodiments, it may bepossible to perform visual ID checks, correct exam verification, performenvironment checks, perform computing device checks, perform examauthorization, perform audio and/or visual observations, and performevent notations. In addition, certain embodiments may solve the issue ofefficiency by only involving a human proctor in the event of a suspectedissue. Using certain embodiments, a live remote proctor could befocusing on six to eight simultaneous online exam sessions that aredesignated as suspect by the system, while hundreds of other users arerunning concurrently.

As for scalability, certain embodiments may solve this problem by, asone example, relying on measured observations that less than about 17%of test-takers will attempt to cheat on an exam. Given, for example, 4million exams, if 17% were to cheat, that would bring the necessaryamount of sessions to observe down to 680,000 that would have beenflagged by certain embodiments of the invention or a proctor. Inaddition, using certain embodiments, an organization would only require109 proctors if the proctors performed full reviews on the sessions thatlasted an average of 2 hours. If the same proctors only reviewed therecorded evidence of cheating on the session, 8 proctors would berequired to successfully proctor 4 million exams.

According to certain embodiments, it may also be possible to address theproblem of wait time in online proctoring. This may be addressed, forexample, by automating the entire exam launching and authorizationsequence according to certain embodiments described herein. In addition,certain embodiments may also resolve issues concerning bandwidthavailability and stability in online proctoring. For instance, accordingto certain embodiments, it may be possible to use high, but minimal losscompression algorithms, along with local storage buffering in order toupload content and events when the bandwidth and connectivity becomebetter.

According to further embodiments, it may be possible to consider andimprove upon user experience, cheater adaptability, and hybridscalability. For instance, with regard to user experience, certainembodiments may work with a large majority of users without the need fordifferent hardware, complex software installations, and drasticmodifications of testing behavior or environment. With regard to cheateradaptability, it may be possible for certain embodiments to, through theuse of machine learning, continue to become more accurate overtime.Further, with regard to hybrid scalability, while certain embodimentsmay operate alone, other embodiments may be designed to work with humancounterparts to augment their ability to perform the proctoringfunction.

In certain embodiments, it may further be possible to provideimprovements in the system's ability for facial recognition matching,lighting checks, and sound level checks. Further, supervised machinelearning may also improve upon pattern detection. According to otherembodiments, it may be possible to integrate video technology withscheduling systems, provide support for continuous identityverification, and provide realtime behavior notifications with respectto normal and/or questionable behavior of test-takers.

One having ordinary skill in the art will readily understand that theinvention as discussed above may be practiced with steps in a differentorder, and/or with hardware elements in configurations which aredifferent than those which are disclosed. Therefore, although theinvention has been described based upon these preferred embodiments, itwould be apparent to those of skill in the art that certainmodifications, variations, and alternative constructions would beapparent, while remaining within the spirit and scope of the invention.In order to determine the metes and bounds of the invention, therefore,reference should be made to the appended claims.

We claim:
 1. An apparatus, comprising: at least one processor; and atleast one memory including computer program code, wherein the at leastone memory and the computer program code are configured to, with the atleast one processor, cause the apparatus at least to: check atest-taker's computing device for compatibility, and for content thatprovide unauthorized aid to the test-taker during a testing session;take a photo of the test-taker; record the test-taker by performing aroom pan while analyzing the surrounding of the test-taker forunauthorized objects; validate the identity of the test-taker by way ofat least one of presenting challenge questions, obtaining voicebiometrics, or obtaining keystroke biometrics; record the test-taker'saudio or video and desktop feed; determine if the test-taker isexhibiting questionable behavior based on an analysis of thecompatibility of the test-taker's computing device, the test-taker'sphoto, the room pan, validation of the test-taker's identity, andrecordation of the test-taker's audio or video and desktop feed; andsuspend the testing session if it is determined that the test-taker isexhibiting questionable behavior.
 2. The apparatus of claim 1, whereinthe at least one memory and the computer program code are furtherconfigured to, with the at least one processor, cause the apparatus atleast to provide the test-taker with test rules defined by aninstitution of the test-taker.
 3. The apparatus of claim 1, wherein theat least one memory and the computer program code are further configuredto, with the at least one processor, cause the apparatus at least tocheck the test-taker's computing device for applications, code, ormaterials used to provide unauthorized aid during the testing session.4. The apparatus of claim 1, wherein the at least one memory and thecomputer program code are further configured to, with the at least oneprocessor, cause the apparatus at least to send the test-taker a warningsignal when it is determined that the test-taker is exhibitingquestionable behavior.
 5. The apparatus of claim 1, wherein the at leastone memory and the computer program code are further configured to, withthe at least one processor, cause the apparatus at least to communicatewith a live proctor when it is determined that the test-taker isexhibiting questionable behavior.
 6. The apparatus of claim 1, whereinthe at least one memory and the computer program code are furtherconfigured to, with the at least one processor, cause the apparatus atleast to: adjust system settings for proctored test sessions; reviewcompleted proctoring test sessions; and view sessions flagged forquestionable behavior of the test-taker.
 7. A method, comprising:checking a test-taker's computing device for compatibility, and forcontent that provide unauthorized aid to the test-taker during a testingsession; taking a photo of the test-taker; recording the test-taker byperforming a room pan while analyzing the surrounding of the test-takerfor unauthorized objects; validating the identity of the test-taker byway of at least one of presenting challenge questions, obtaining voicebiometrics, or obtaining keystroke biometrics; recording thetest-taker's audio or video and desktop feed; determining if thetest-taker is exhibiting questionable behavior based on an analysis ofthe compatibility of the test-taker's computing device, the test-taker'sphoto, the room pan, validation of the test-taker's identity, andrecordation of the test-taker's audio or video and desktop feed; andsuspending the testing session if it is determined that the test-takeris exhibiting questionable behavior.
 8. The method of claim 7, whereinthe method further comprises providing the test-taker with test rulesdefined by an institution of the test-taker.
 9. The method of claim 7,wherein the method further comprises checking the test-taker's computingdevice for applications, code, or materials used to provide unauthorizedaid during the testing session.
 10. The method of claim 7, wherein themethod further comprises sending the test-taker a warning signal when itis determined that the test-taker is exhibiting questionable behavior.11. The method of claim 7, wherein the method further comprisescommunicating with a live proctor when it is determined that thetest-taker is exhibiting questionable behavior.
 12. The method of claim7, wherein the method further comprises: adjusting system settings forproctored test sessions; reviewing completed proctoring test sessions;and viewing sessions flagged for questionable behavior of thetest-taker.
 13. A computer program, embodied on a non-transitorycomputer readable medium, the computer program, when executed by aprocessor, causes the processor to: check a test-taker's computingdevice for compatibility, and for content that provide unauthorized aidto the test-taker during a testing session; take a photo of thetest-taker; record the test-taker by performing a room pan whileanalyzing the surrounding of the test-taker for unauthorized objects;validate the identity of the test-taker by way of at least one ofpresenting challenge questions, obtaining voice biometrics, or obtainingkeystroke biometrics; record the test-taker's audio or video and desktopfeed; determine if the test-taker is exhibiting questionable behaviorbased on an analysis of the compatibility of the test-taker's computingdevice, the test-taker's photo, the room pan, validation of thetest-taker's identity, and recordation of the test-taker's audio orvideo and desktop feed; and suspend the testing session if it isdetermined that the test-taker is exhibiting questionable behavior. 14.The apparatus of claim 13, wherein the computer program, when executedby the processor, further causes the processor to provide the test-takerwith test rules defined by an institution of the test-taker.
 15. Theapparatus of claim 13, wherein the computer program, when executed bythe processor, further causes the processor to check the test-taker'scomputing device for applications, code, or materials used to provideunauthorized aid during the testing session.
 16. The apparatus of claim13, wherein the computer program, when executed by the processor,further causes the processor to send the test-taker a warning signalwhen it is determined that the test-taker is exhibiting questionablebehavior.
 17. The apparatus of claim 13, wherein the computer program,when executed by the processor, further causes the processor tocommunicate with a live proctor when it is determined that thetest-taker is exhibiting questionable behavior.
 18. The apparatus ofclaim 13, wherein the computer program, when executed by the processor,further causes the processor to: adjust system settings for proctoredtest sessions; review completed proctoring test sessions; and viewsessions flagged for questionable behavior of the test-taker.